feat: run gateway tool loop natively on /v1/messages#131
Conversation
76d4433 to
f7a7759
Compare
69fffd3 to
229070d
Compare
229070d to
6116f55
Compare
When an Anthropic Messages request declares a gateway-owned tool (web_search), run the server-side gateway tool loop natively against vLLM /v1/messages — execute the tool, hide it from the client, and surface only the final assistant message. Requests without a gateway-owned tool keep the transparent proxy (vllm-project#99). The loop talks Anthropic Messages end to end (no RequestPayload/ResponsePayload detour), so the client's request is forwarded to vLLM untouched and every Anthropic field is preserved. It reuses only the protocol-neutral tool layer (ToolRegistry::dispatch, name-based classification, per-call timeout, concurrent execution) via a small tool seam. - types/messages: Anthropic wire types + tool seam (tool_use<->FunctionToolCall, ToolOutput->tool_result, gateway/client classification). - executor::messages_loop: non-streaming loop (hide-the-call, feed tool_result back, round cap; client-owned tool_use returns to the client). - executor::messages_stream: streaming loop + MessagesStreamAccumulator that presents one logical message across rounds (single lifecycle, contiguous block indices, gateway tool_use suppressed, client-owned tool_use terminal) — the Anthropic-native analogue of the Responses GatewayAccumulator (vllm-project#119). - handler: routes gateway-tool requests to the loop, forwards the client's x-api-key, returns the Anthropic error envelope on failure. Tests: tool-seam + accumulator unit tests; non-streaming and streaming acceptance tests replaying live-recorded /v1/messages cassettes (single-round, parallel, sequential multi-round, no-tool-call); edge coverage for tool failure, upstream error, round cap, malformed input, and mixed client+gateway tools; handler routing tests. Validated live on G6e (Qwen3 + real You.com) via curl and a real upstream Claude Code CLI session. Implements vllm-project#115. Signed-off-by: Ashwin Giridharan <girida@amazon.com>
6116f55 to
28a989c
Compare
maralbahari
left a comment
There was a problem hiding this comment.
@ashwing thanks for the PR. I added some inline comments
Address review on the Messages-native gateway tool loop:
- Feed the model's full assistant turn (thinking/text/signature +
tool_use, order preserved) into the next round's history in both the
streaming and non-streaming loops, instead of reconstructing it from
only the gateway tool_use blocks. Dropping the preceding blocks lost
conversation state and could invalidate extended-thinking follow-ups.
- Reject malformed/incomplete tool input (streaming partial_json or a
non-object input) with an is_error tool_result rather than coercing to
{} and dispatching the tool with arguments the model never supplied.
- Apply a consistent mixed-call policy: when an assistant turn contains
both a gateway-owned and a client-owned tool_use, hide the gateway
call, surface the client call, and stop the loop -- in both paths.
- Reset final_message_delta in begin_round() so a clean-EOF round cannot
re-emit the previous round's stop_reason: tool_use.
Adds repro tests covering each case in both loops.
Signed-off-by: Ashwin Giridharan <girida@amazon.com>
| /// Gateway-tool execution failures do **not** error — they become error | ||
| /// `tool_result`s fed back to the model. | ||
| pub async fn run_messages_loop( | ||
| mut request: Value, |
There was a problem hiding this comment.
why instead of raw json value not using MessagesRequest that you have in types/messages.request.rs ?
There was a problem hiding this comment.
@ashwing if the raw lossless json is really required maybe consider passing a per-request context containing both the typed MessagesRequest for safe field access and the raw JSON Value for lossless upstream forwarding? or you can follow the pattern we use for RequestPayload for Responses API where it implements to_upstream function. so wouldnt need to maintain a context where it holds lossless for upstream and typed version for gateway.
There was a problem hiding this comment.
Fair question. The difference from RequestPayload::to_upstream_request is that that path transforms — it flattens Codex namespace tools, normalizes every tool type to Vec<FunctionTool>, resolves tool_choice. Typed→typed is the actual work there. This loop is a pass-through: it forwards the client's request to vLLM /v1/messages verbatim and only reads tools + rewrites messages/stream, so there's no transform step a typed request would carry.
One concrete catch with going fully typed here: ContentBlock has #[serde(other)] Unknown, so deserializing→reserializing through MessagesRequest would silently drop any block type we don't model (redacted_thinking, future block types, provider extensions). Raw Value forwards those untouched, which matters for a proxy.
That said — you're right that a per-request context (typed MessagesRequest for safe field access + raw Value for lossless forwarding) is the cleaner long-term shape, and it lines up with the loop-consolidation follow-up already noted in the PR description. I'd rather not reshape the request plumbing inside this PR (it'd touch the routing path and widen the diff), so I'll file it as a follow-up and link it here. Sound good?
There was a problem hiding this comment.
@ashwing thanks ashwin, you can create Issue for this for the follow up. approving this PR.
Records the plan to classify gateway-owned tools by shape (not name) on the Messages side so Claude Code's web search and MCP can run through the gateway, mirroring the Responses side. Grounded in live testing: Codex declares web search structurally and already routes through the loop; Claude Code declares a client-owned WebSearch function that proxies. A name alias was rejected as a client-tool hijack. Follow-up to vllm-project#131. Signed-off-by: Ashwin Giridharan <girida@amazon.com>
58fa6bb to
83ca75b
Compare
…e Code) Claude Code declares its built-in web search as a client function named `WebSearch`, so it fell through the name-based `web_search` classifier and was proxied — Claude Code's web search never ran server-side, unlike Codex, whose typed `web_search` tool routes through the Responses loop. Add an operator-configured client-tool -> gateway-executor alias map (`MESSAGES_GATEWAY_TOOL_ALIASES`, e.g. `WebSearch=web_search`), empty by default. This is the ownership doctrine's "client function is client-owned unless configured as gateway-owned" clause (docs/design/codex-integration.md), applied structurally rather than by a hardcoded name. When configured: - routing, registry build, and both loops classify the aliased tool as gateway-owned via GatewayToolMap; - dispatch canonicalizes the call name to the `web_search` executor key AND adapts the arguments (allowed_domains -> include_domains, blocked_domains -> exclude_domains) so Claude Code's WebSearch schema reaches the executor correctly; - the raw request still forwards to vLLM verbatim, so the model sees the client's own tool contract; the gateway executes and hides the call. Adds the follow-up design doc and unit tests for classification, env parsing, name canonicalization, and the argument adapter. Signed-off-by: Ashwin Giridharan <girida@amazon.com>
Part of #113; implements #115 (Stage 2).
What
Run the gateway tool loop natively on
/v1/messages: when a request declares a gateway-owned tool, the gateway executes it server-side, hides it, and returns only the final assistant message. No gateway tool → unchanged transparent proxy.This also makes Claude Code's web search run on the gateway — the Codex experience, for Claude Code.
How
The loop is native to Anthropic Messages — it talks vLLM
/v1/messagesupstream and forwards the client's request untouched, so every Anthropic field (tool_choice,stop_sequences, thinking, …) is preserved. It reuses only the protocol-neutral tool layer (ToolRegistry::dispatch) via a small seam; it never touchesRequestPayload/ResponsePayload.types/messages/tool_seam—tool_use↔FunctionToolCall,ToolOutput→tool_result, gateway/client classification.executor::messages_loop— non-streaming loop (hide-the-call, feedtool_resultback, round cap).executor::messages_stream— streaming loop + accumulator presenting one logical message across rounds (single lifecycle, contiguous block indices, gatewaytool_usesuppressed).handler— routes gateway-tool requests to the loop; Anthropic error envelope on failure.Claude Code web search
Claude Code declares web search as a client tool named
WebSearch(not a typed server tool like Codex'sweb_search), so it is client-owned by default. Operators opt in withMESSAGES_GATEWAY_TOOL_ALIASES="WebSearch=web_search"(empty by default) to have the gateway execute it — the ownership doctrine's "client function is client-owned unless configured as gateway-owned" (codex-integration.md). Dispatch canonicalizes the name to theweb_searchexecutor and adapts the arguments (allowed_domains→include_domains,blocked_domains→exclude_domains). See the README's Claude Code section anddocs/design/messages-gateway-tool-classification.md.Tests
cargo fmt --check/clippy --workspace --all-targets -D warnings/cargo test --workspace— clean./v1/messagescassettes — single-round, parallel, sequential multi-round, no-tool-call, streaming.tool_result, upstream error → Anthropic envelope, round cap, malformed args, mixed client+gateway tools, proxy fallthrough.web_search) and real Claude Code CLI (aliasedWebSearch) both drove the loop server-side with the call hidden — routedmessages_loop, live search backend hit, single streaming lifecycle, no raw-Responses leakage.Known limitations (intentional)
pause_turnsurfaced, not continued — onlystop_reason: tool_usedrives another round; other stops return to the client.WebSearchdomain filters — the executor treats include/exclude domains as mutually exclusive, so a call setting bothallowed_domainsandblocked_domainsreturns a graceful error result.Follow-up: consolidation with the Responses loop
This adds a second gateway-tool loop alongside the Responses one; the duplication is deliberate (different wire shapes; the typed core shouldn't be reshaped ahead of the Layering ADR). Tool execution is already shared. Planned follow-ups (post-ADR): a
GatewayTurntrait + one generic loop, a shared stream-normalizer contract, hoisted shared limits, and structural typed-tool + per-request opt-in classification on the Messages side (extends the sameGatewayToolMapseam).